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[57] ABSTRACT 

An automatic testing method and device is described that 
can test a video sequence coder/decoder system and either 
assess the quality of decoded sequences or rate the fidelity 
of the coding chain. The device produces synthetic test 
patterns that induce the appearance of known artifacts, then 
tracks and evaluates such artifacts. Based on this evaluation, 
it can rate the system's performance in a way that correlates 
well with human assessments. In our testing device, the 
quality estimation module performs this function. 

18 Claims, 9 Drawing Sheets 
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TESTING ARCHITECTURE FOR DIGITAL 
VIDEO TRANSMISSION SYSTEM 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to digital video transmission 
systems and, more particularly, to testing digital video 
transmission systems. 

2. Description of the Related Art 

As the market and number of products for multimedia 
applications grow, the need for testing methods and fidelity 
metrics becomes critical. Testing devices are important for 
the development and evaluation of multimedia products, for 
example, to assess MPEG coding fidelity. 

It is important to test digital coders/decoders during either 
their development or installation. However, assessing pic- 
ture quality in the digital domain is a problem because it 
involves human perception. The most commonly used 
metric, the signal-to-noise ratio, does not match human 
judgment well. See, for example, P. M. Farrelle, Recursive 
Block Coding for Image Data Compression, Springer 
Verlag, New York, 1990. 

Furthermore, conventional digital transmission testing 
systems typically transmit and evaluate natural scenes. This 
approach has several disadvantages. Natural scenes are 
resolution dependent and hence are difficult to produce with 
varying frame rate or size. Furthermore, complex natural 
scenes can require a large amount of memory to store and 
provide little latitude in testing the features of a particular 
coder due to the inflexibility of these scenes. Moreover, the 
test patterns are not deterministic (that is, we do not know 
the equations producing them), which makes quality evalu- 
ation difficult. 

Thus, it can be seen that conventional digital testing 
systems using natural scene testing techniques impose limits 
upon current digital video testing devices, and hinder the use 
of these devices in many applications. 

Therefore, there is an unresolved need for an automatic 
testing device that can test a video sequence coder/decoder 
system to assess the quality of decoded sequences and rate 
the fidelity of the coding chain in a way that correlates well 
with human assessments. 

SUMMARY OF THE INVENTION 

An automatic testing method and device is described that 
can test a video sequence coder/decoder system and either 
assess the quality of decoded sequences or rate the fidelity 
of the coding chain. The device produces synthetic test 
patterns that induce the appearance of known artifacts, then 
tracks and evaluates such artifacts. Based on this evaluation, 
it can rate the system's performance in a way that correlates 
well with human assessments. In our testing device, the 
quality estimation module performs this function. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention will be readily understood by the following 
detailed description in conjunction with the accompanying 
drawings, wherein like reference numerals designate like 
structural elements, and in which: 

FIG. 1 is a block diagram illustrating a testing architecture 
embodiment according to the present invention; 

FIG. 2 illustrates a synchronization frame containing 
synchronization code and customization information as 
practiced according to one embodiment of the present inven- 
tion; 
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FIG. 3 illustrates the third and thirtieth frames of an edge 
rendition test sequence as practiced according to one 
embodiment of the present invention; 

FIG. 4 illustrates the third and thirtieth frames of a 
5 blocking-effect test sequence as practiced according to one 
embodiment of the present invention; 

FIG. 5 illustrates the third and thirtieth frames of an 
isotropy test sequence as practiced according to one embodi- 
ment of the present invention; 

FIG. 6 illustrates the third and thirtieth frames of an edge 
rendition test sequence as practiced according to one 
embodiment of the present invention; 

FIG. 7 illustrates complex texture frames used to fill the 
j 5 video buffer verifier according to one embodiment of the 
present invention; 

FIG 8 illustrates the eighth and thirty fifth frames of a 
blocking-effect test sequence into which five frames of 
complex textures have been inserted immediately after the 
20 synchronization frame as practiced according to one 
embodiment of the present invention; and 

FIG. 9 illustrates a block diagram of quality metrics 
according to one embodiment of the present invention. 

25 DETAILED DESCRIPTION OF THE 

INVENTION 

Embodiments of the invention are discussed below with 
reference to FIGS. 1-9. Those skilled in the art will readily 
30 appreciate that the detailed description given herein with 
respect to these figures is for explanatory purposes, however, 
because the invention extends beyond these limited embodi- 
ments. 

We describe herein an automatic testing device that can 
35 test a video sequence coder/decoder system and either assess 
the quality of decoded sequences or rate the fidelity of the 
coding chain. The device produces synthetic test patterns 
that induce the appearance of known artifacts, then tracks 
and evaluates such artifacts. Based on this evaluation, it can 
40 rate the system's performance in a way that correlates well 
with human assessments. In our testing device, the quality 
estimation module performs this function. 

System Structure 

45 We base our test methodology on a library of synthetic 
test patterns, which offer several advantages over natural 
scenes: 

Algorithmically generated synthetic patterns are resolu- 
5Q tion independent and hence can be generated with any 
frame rate or size. 
Synthetic patterns require much less memory than com- 
plex natural scenes. 
Algorithms can be designed to generate customizable 
55 patterns, which provide more latitude in testing the 
features of a particular coder. 
Test patterns are entirely deterministic (that is, we know 
the equations producing them), which makes quality 
evaluation easier. 
60 Two key questions are associated with such a procedure: 
1) Is a procedure that uses synthetic test patterns a valid test 
procedure? and 2) Can synthetic scenes accurately test a 
digital-image coder designed to process natural scenes? 
This test methodology is valid provided the synthetic 
65 sequence, when presented to the test device, has a behavior 
comparable to that of natural sequences. This implies that 
the test pattern scenes have the same coding complexity and 
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basic features as the natural images. That is to say, the For one embodiment, every bit (i.e., every pixel of the 

synthetic scene should produce the same amount of distor- synchronization frame) is replicated several times in both 

tion as a natural scene and should contain contours or the horizontal and vertical directions to recover error-free 

textures that resemble natural elements. We imposed this synchronization data. The number of repetitions is a function 

requirement when designing each test sequence. 5 of ^ dimension. Thus, if the encoder 130 is scalable, this 

FIG. 1 is a block diagram depicting a testing system, . , U1 , . J r c 4l _ . 
which consists of emitter-encoder and decoder-receiver av0lds P roblems due t0 downsamphng of the unages. 
tandems, as practiced according to one embodiment of the For one embodiment, the synchronization 160 part con- 
present invention. One can view the testing device as tains two length-13 Barker codes. We chose this word length 
consisting of two modules, namely the emitter 100 and the J0 t0 be robust tQ channd errof Modules afe synchronized 
receiver 170. Emitter 100 and receiver 170 are coupled to wheQ ^ ^ no receives hronization code ^ 
the respective head and tail end of coding chain 120, which - . e . ' . , 
includes encoder 130, channel 140 and decoder 150. For one fewer than four bit errors. This yields a correct-locking 
embodiment, the encoder 130 and decoder 150 are MPEG- probability of 1.0-10" and a 1.69x10-* false-locking 
based and are external to the testing device. Alternately, the probability, 
encoder 130 and decoder 150 are internal to the tester. For 15 

an embodiment wherein encoder 130 is tested, the channel Customizing Test Sequences 
140 and decoder 150 are internal to the tester. Similarly, for 

an embodiment wherein decoder 150 is tested, the decoder With this testing device, users have considerable flexibil- 

150 and channel 140 are internal to the tester. For yet another j ty m specifying the test sequences. For instance, users can 

alternate, the encoder 130 and decoder 150 operate accord- 20 specif y which test {Q mn md me test dimensions> 

ing to a non-MPEG protocol. can also define a suite of tests> ix>> a new test tnat 

Tne emitter 100 includes a pattern generator 110, while combines other existing tests or suites . Furthermore, for 

the receiver 170 includes the quality evaluation module 190 ^ or gui usefS can tf how man times ^ 

and a pattern generator 180 able to produce patterns similar ^ forms ^ ^ cedurc> ^ also ^ 

to those produced by the pattern generator 110 (of the 25 j. *• f J r / 

emitter 100). Thus, the testing device has access to the h * °f the * use for quahty evaluation. Table 

original and decoded versions of the test sequence from 1 hst f ^ stmctl f of ' he synchronization frame which 

which the quality evaluation module 190 can make quanti- completely specifies the test to be performed, for one 

tative measurements regarding image fidelity. embodiment of the present invention. 

Because both pattern generators 100 and 180 preferably 30 

generate the same sequences concurrently to estimate TABLE 1 

quality we have designed a synchronization procedure and structure of the test synchronization fmme. 
inserted synchronization information 160 mto the test pat- 
tern. Furthermore, the pattern generator 180 in the receiver 

170 must know which pattern the pattern generator 110 of 35 
emitter 100 generated. The test pattern also includes pattern 
related-information, which the receiver module 170 extracts 
from the decoder 150 output. 

Synchronization ^ 

We use Barker codes to synchronize the two test pattern 
generators located at two different sites. Alternate synchro- 
nization schemes are known in the art, however, and may be 
employed. Barker codes are useful for synchronization 
because their autocorrelation sequence has a value of one for 45 
any lag not equal to zero and a value equal to the code length 

for a zero lag (see, for example, M. S. Rodeo, Digital FIG. 2 presents an example of this synchronization frame, 

Communication Systems Design, Prentice Hall, Englewood wmch looks ^ a barcode image. The synchronization 

Clifife, N.J., 1988.). The emitter 100 communicates synchro- frame of FIG - 2 contains the synchronization code and the 

nization and pattern customization information to the 50 customization information, 
receiver 170, because the testing device considers the whole 

coding chain 120 (encoder 130-channel 140-decoder 150) to Tested Features 

be a black box. 

Furthermore, for an embodiment wherein we may not Table 2 contains a list of tested features for one embodi- 
manipulate the encoder 130 output bitstream, we insert the 55 ment of the present invention. For this embodiment, we 
data into images of the sequence as follows. The first frame intended this testing device to test an MPEG coding chain 
of the test sequence, called the synchronization frame, from the outside, i.e., from an end -user viewpoint (e.g., 
contains the desired synchronization and customization data. broadcasters, television equipment manufacturers, and net- 
Luminance values represent bits of data; typically, the work providers). Hence, we did not test the bitstream syntax, 
lowest possible value represents a 0 and the highest value a 60 For instance, one embodiment of the testing device cannot 
1 . For one embodiment, these values are 16 and 235 accord- determine if the variable-length code (VLQ conforms to the 
ing to Comite* Consulatif International des Radio Commu- MPEG specifications. However, alternate embodiments hav- 
nications (CCIR) Recommendation 601. See, CCIRRecom- ing the additional ability to test bit stream syntax can be 
mendation 601-1, Digital Television Coding Parameters for produced. For example, P. H. Meehan et al. have described 
Studio Recommendations and Reports of the Consultative 65 a bitstream syntax test procedure. See, R H. Meehan et al„ 
Committee for Radiocommunications t Vol. XI- 1, pp. "MPEG Compliance Bitstream Design", Proc. IEEE Con- 
319-328, 1986. swner Electronics Soc, IEEE, Piscataway, NJ., 1995. 



Component 


Size (bits) 


Synchronization code 


32 


Image X-size 


16 


[mage Y-size 


16 


Test number 


16 


Test suite number 


16 


Test repeat number 


16 


Number of frames in test 


16 


Active portion 


4x 16 


Other parameters or future extensions 


144 


Total 


336 
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Returning to Table 2, each feature of this embodiment is 
listed, along with the test pattern that tests the feature. Some 
of these patterns mimic test procedures for analog television 
devices, while algorithms used in digital image coding 
inspire other patterns. 

TABLE 2 

Features currently supported by the test pattern generator and 
corresponding test patterns. 



Feature 



Test Pattern 



Luminance rendition 
Chrominance rendition 
Edge rendition 
Blocking effect 
Isotropy 

Abrupt scene changes 
Noise test 

Text rendition 
Texture rendition 
Tune aliasing 
Tracking, motion 



Buffer control 



Moving luminance bars 

Moving color bars 

Rotating square and moving circle 

Chessboard pattern of diamonds 

Moving circular zone plate 

Scene with abrupt temporal changes 

Still image with increasing noise as a 

function of time 

Moving and still text 

Animated texture 

Rotating circular wheel 

Circular-moving zone plate and 

rotating- wheel pattern plus contour 

rendition with zooming, panning at 

variable speed, addition of noise 

Complex images inserted in a test 

sequence 



10 



15 



20 



25 



Tested Features 

Luminance and chrominance rendition tests consist of 30 
colored bars that move. In this way, we can examine color 
representation and possible color clipping. 

By examining the influence of quantization and "mos- 
quito noise" (phase ambiguity of the DCT, direct cosine 
transform), the edge rendition test looks at edge rendition as 35 
a function of orientation. For these purposes, the test 
sequence alternates between images of a rotating square and 
a moving circle. Both objects change color over time and 
their contours are antialiased. 

The blocking effect test examines the appearance of 40 
blocking artifacts on a sequence of squares whose size 
decreases with time as the, number of squares increases. 
Each square is filled with a pattern whose luminance varies 
as a function of its spatial coordinates. 

The isotropy test determines whether the coder is isotro- 
pic. This can be observed for either intra-coded or tempo- 
rally predicted frames. The ideal test pattern is a circular- 
zone plate. 

The abrupt scene changes test determines how well the 
coder copes with difficult scene changes, while the noise test 
examines the influence of increasing noise (as a function of 
time) on a still picture. 

Block-DCT tends to have a disastrous effect on small 
fonts. Therefore, the text rendition test checks the coder's 55 
performance on text of various size. 

The texture rendition test presents a time-varying texture 
to the coder. 

To exhibit artifacts arising from poor motion estimation 
and compensation, we use the time-aliasing and motion 60 
rendition tests. They test both linear and nonlinear motion at 
various speeds and also simulate zooming and panning. 
These tests use circular-zone plate and rotating-wheel test 
patterns. 

We can evaluate the performance of the bit rate regulation 65 
algorithm in constant bit rate coders using the buffer test. In 
constant bit rite coders, the buffer's fullness strongly influ- 



45 



50 



ences an artifacts visibility. Hence, to evaluate the buffer, 
we test artifacts at various buffer occupancy levels. For one 
embodiment, the synthetic pattern for the buffer test consists 
of up to five texture images that are very complex to code. 
Inserting some of these in other test sequences fills the buffer 
to a certain level and allows us to study the behavior of the 
resulting artifacts. 

Implementation 

For one embodiment, our test pattern generator consists of 
software running on a Unix system. The test pattern gen- 
erator can generate all of the test sequences described earlier, 
and users can fully customize the sequence. Output can be 
in any chroma format (luminance only, 4:2:0, 4:2:2, 4:4:4), 
and users can choose between interlaced or progressive 
formats. The test pattern generator can also generate motion 
at either full-, half-, or fractional-pixel accuracy. 

Some of the test patterns comprise contours. Because a 
contour is by definition a high-frequency signal, sampling it 
is a delicate operation if we want a good visual impression. 
It was thus necessary to somehow emulate the low-pass 
filtering characteristics of acquisition devices by using anti- 
aliasing techniques, which yield smooth contours devoid of 
the well-known staircase effect. 

For one test pattern generator embodiment, we use a 
two-point antialiasing method. See, X. Wu, "An Efficient 
Antialiasing Technique", Computer Graphics, Vol. 25, No. 
4, July 1991, pp. 143-152. Another stage adds a gamma 
correction of the computed pixel value to eliminate the 
Moire pattern that can appear with antialiasing techniques. 
To meet the system's low-cost and real-time implementation 
requirements, the test pattern generator incorporates various 
optimizations of the pattern generation algorithms. First, we 
use incremental techniques and exploit pattern symmetries 
to obtain the sampling (i.e., rendering) of drawing primi- 
tives. In the worst case, rendering a primitive requires only 
a few additions per pixel. In instances requiring fractional 
arithmetic, we used 16-bit fixed-point arithmetic instead of 
the more expensive floating-point arithmetic. 

Examples of Test Patterns 

In FIGS. 2 through 8 we present some test patterns and the 
resulting encoding/decoding process as performed by an 
embodiment using an MPEG-2 coding scheme. 

Sequences and Encoding Parameters 

We generated all sequences shown in FIGS. 2 through 8 
in CIF format (352x488, 30 Hz). They have been encoded 
and decoded using an MPEG-2 software simulator (provided 
by the MPEG Software Simulation Group, C. Fogg, 
"mpeg2encode/mpeg2decode", version 1.1, available by via 
web site at HTTP://www. mpeg.org/index.html/, MPEG 
Software Simulation Group, June, 1994) at main profile and 
main level. The encoding parameters were: 

Bit rate: 750 Kbits/sec 

Group of pictures dimension: 12 frames 

I/P frame distance: 3 frames 

Video buffer verifier size: 480 Kbits 

Defaults quantization matrices 

Progressive sequence 

P-frame search window for motion estimation: 15 pixels 
B-frame search window for motion estimation: 7x7 pixels 

Edge Rendition 

Several frames of the synthetic test patterns used for the 
edge rendition test are shown in FIG. 3. Specifically, frames 
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three (top left) and thirty (top right) of the edge rendition test broadcasters, equipment manufacturers, application 

sequence are shown in FIG. 3, with the decoded frames designers, and network providers in testing decoders, encod- 

shown below the originals. This test pattern features a ers and transmission channels. 

rotating square that changes color over time. The test We have developed a design for the quality evaluation 

displays the square on a nonuniform background generated 5 module 190. The generic quality measure of signal-to-noise 

by a slowly varying function of the luminance value to rat i 0 is well known to be uncorrectable with human visual 

which we add light white noise. The figure shows original evaluation and is thus not preferred for use with our device, 

and decoded versions of the frames. Contours exhibit some prior work in this field concentrates on still-picture quality 

distortion due to coding; commonly called "Mosquito evaluation. See, P. M. Farrelle, Recursive Block Coding for 

noise", this distortion is due to the phase ambiguity of the 10 image Data Compression, Springer Verlag, N.Y., 1990; J. L. 

OCT. Manos and D. J. Sakrison, "The Effects of a Visual Fidelity 

Criterion on the Encoding of Images", IEEE Trans. Infor- 

Blocking Effect mation Theory, Vol. IT-20, No. 4, 1974, pp. 525-536; J. A. 

Hie blocking-effect test sequence features a series of 1C Sagbri, p s - Cheatham, and A. Habibi, "Image Quality 

squares whose dimensions decrease with time. Specifically, 15 Measure Based on a Human Visual System Model", Optical 

frames three (top left) and thirty (top right) of the blocking- VoL 3 No> 1 l} 98 ?' JP f. 13 "* 1 * ? nd , S * 9? me * 

effect test sequence are shown in FIG. 4, with the decoded B - Mac <l> Human Vlsual Q uaht y Criterion \ SPIE Visual 

frames shown below the originals. Each square is filled with Comm. and Image Processing, Vol. 1360, 1990, pp. 2-7. 

a slowly varying function of luminance that favors the Based on the test pattern generator, we have developed 

appearance of the blocking effect due to the use of the block numerical metrics that parameterize distortions and account 

DCT. As sequence complexity increases, the blocking for human perception. For this purpose, we have developed 

effect's importance increases as well. a model of the human visual system. See: Christian J. van 

den Branden Lambrecht and Olivier Verschuere, "Perceptual 

Isotropy Quality Measure using a Spatio-Temporal Model of the 

, . , , , . L 25 Human Visual System", Proc. of the SPIE, vol. 2668, pp. 

Avery well-known and widely used pattern for testing the 45(M61> SPIE> S an Jose, Calif., Jan. 28-Feb. 2, 1996; 

isotropy of display or coding equipment is the zone p ate. Chistian j yafl den Branden « A Working Spatio- 

FIG. 5 shows two frames of the moving zone plate Te d Modd of ^ Human vifiual g fof j 

Specifically, frames three (top left) and thirty (top right) of Rest0 r a tion and Quality Assessment Applications", Proc. 

the isotropy test sequence are shown m FIG. 5, with the 30 /#|f7 Qonf AcousticSf s h , and Signal Processing, pp. 

decoded frames shown below the originals. The top-nght 2293 _ 2296) IEEE> piscataway, N.J., May 7-10, 1996; and 

zone plate is at a higher frequency than the top-left. We add c j van deQ Branden Lambrechtj Perceptua i Models and 

white noise to the plate to analyze the coder isotropy in the Architectures for Video Coding Applications, PhD Thesis, 

presence of noise. Swiss Federal institute of Technology, CH 1015 Lausanne, 

Motion Rendition 35 Switzerland, September, 1996. Each of these documents 

may be obtained via web site at HTTP://ltswww.epfl.ch/ 

This test is one of the most complex, because motion pub_Jiles/vdb/. 
estimation is such a key feature of video coding. We divide 

the test into two main subsequences. The first tests the Metrics 

rendition of translational motion and uses the same zone sec tion describes metrics suitable for use in evalu- 

plate as the isotropy test. We augment testing by simulating ation mo dule 190 of FIG. 1. It explains how using the above 

zooming and panning on the plate. The second subsequence, referenced knowledge about the human visual system, a 

illustrated in FIG. 6, tests the rendition of non-translational mo fei suitable for image processing purposes is built. The 

motion on a rotating wheel. The wheel's center is initially general structure of the model is presented in FIG. 9 and the 

still, but then moves as the test simulates zooming and various building blocks are described hereafter. 

pannmg ' Thus, FIG. 9 is a block diagram of the quality metrics for 

Buffer Control an em ' )0( *" neDt °f me evaluation module 190 of FIG. 1. The 

thick lines represent a set of perceptual components. The 

As explained previously, the buffer test has a structure 50 thin lines represent sequences. The steps involved in the 
different than that of the other tests. It consists of from one computation of the metric are the following: first of all, a 
to five frames of complex textures (FIG. 4 illustrates four of coarse segmentation (930) of the original sequence is com- 
them) that we can insert at the beginning of any other test. puted. The original and coding error sequences are decom- 
Encoding of such complex frames requires a large number posed into perceptual components by tie filter bank (900 
of bits, and a constant bit rate MPEG-based system only 55 and 910). Contrast sensitivity and masking (920) are con- 
meets the bit rate requirement over a full group of pictures sidered in each of the channels. This is done by computing, 
(GOP). Thus, including such frames at a critical position pixel by pixel, the actual detection threshold, considering 
(such as the beginning of a GOP) causes the output buffer to that the original sequence will be a masker to the distortion, 
fill up quickly and forces a much coarser quantization of the The masking strategy then consists of dividing the filtered 
subsequent frames (FIG. 8). Specifically, frames eight (top 60 error signal by the detection threshold. This expresses the 
left) and thirty-five (top right) of a blocking-effect test data in units above threshold also referred to as just notice - 
sequence are shown in FIG. 8, with the decoded frames able differences (j n d's). Finally, the data are pooled (940) 
shown below the originals. This effect worsens as the buffer over the channels to compute the distortion measures, 
size decreases. For one embodiment, the segmentation that is used seg- 

In our methodology, we do not simply check the bitstream 65 ments the sequence into uniform areas, contours, and tex- 

syntax. Rather, our device will perform an end-to-end test in tures. Basically it proceeds as follows: the input images are 

the pixel domain. We intend our device for use by parsed and processed block by block. Hie variance of each 
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elementary block is computed, as well as the variance in the 
horizontal, vertical, and diagonal directions. This data will 
be used to assess the nature of the center pixel of the block. 
If the global variance is below some threshold, the pixel is 
considered as belonging to a uniform area. If the variance is 5 
above the threshold and more or less isotropic, the pixel is 
considered to be lying on a contour. 

Masking the output of the perceptual channels predicts the 
response from the cells of area VI of the cortex. The data has 10 
then to be gathered together to account for higher level of 
perception, which is termed pooling. This work proposes a 
multi-measurement scheme, into which not only a global 
quality measure is computed for a sequence but some 
detailed metrics as well. For one embodiment, a global 
quality metric is calculated, as well as more detailed metrics 
that measure rendition of three basic components of images; 
uniform areas, contours, and textures. Pooling will be 
slightly different for the global metric than for the detailed 
metrics. 



N, N x Ny V 

kl*. J. C]\ 



(1) 



15 



20 



Human observers are never looking at the whole picture 
at the same time but rather at regions. This is due to both the 
focus of attention and the viewing distance. To take those 
facts into account, the global metric can be calculated over 25 
blocks of the sequence. Such blocks are three-dimensional 
and their dimensions are chosen as follows: the temporal 
dimension is chosen to consider focus of attention, i.e., the 
size is computed so that a block covers two degrees of visual 
angle, which is the dimension of the fovea. The distortion 
measure is computed for each block by pooling the error 
over the channels. Basically, the magnitude of the channels' 
output are combined by Minkowski summation with a 
higher exponent to weight the higher distortion more. The 
actual computation of the distortion E for a given block is 
computed according to eq. (1): 
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where e (x, y, t, c) is the masked error signal at position (x, 
y) and time t in the current block and in the channel c; N^, 
Ny, and N, are the horizontal and vertical dimensions of the 
blocks; N is the number of channels. The exponent of the 
Minowski summation is p and has a value of 4, which is 
close to probability summation. See Andrew B. Watson, 
Handbook of Perception and Human Performance, Vol. 1, 
Sensory Processes and Perception, Chapter 6, Temporal 
Sensitivity, John Wiley, 1986. This metric is termed moving 
picture quality metric (MPQM). 

Pooling for the detailed metrics is slightly different. Once 55 
the masked error signal is known, a Minkowski summation 
is directly computed region by region (for textures, uniform 
areas, contours). This directly constitutes the distortion 
measure for each of the features. 

The distortion E computed in Eq. (1) has to be expressed 60 
in some known scale to be significant. Because engineers are 
used to working with decibels (dB's), the distortion can be 
expressed on a logarithmic scale. A metric, that by analogy 
to the work of Comes, see Serge Colmes, Les traitements 
perceptifs d' images numeristes, PhD thesis, University 65 
Catholique de Lovain, 1995, can be named MPSNR 
(masked peak signal-to-noise ratio), is then computed as: 
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MPSNR = lOlogiQ— . 



This scale does not have the exact same meaning as the 
known dB's, hence we refer to it as "visual decibels" 
(vdB's). Alternately, one can use a quality scale that has 
often been used for subjective testing in the engineering 
community. See, e.g., M. Ardito, M. Barbero, M. Stroppiana, 
and M. Visca, "Compression and Quality", In L. 
Chiariglione, editor, Proceedings of the International Work- 
shop on HDTV 94, pp. B-8-2, Torino, Italia, Oct., 26-28, 
1994, Springer Verlag; and S. Comes and B. Macq, "Human 
Visual Quality Criterion", SPJE Visual Comm, and Image 
Processing, Vol. 1360, 1990, pp. 2-7. It is a scale from 1 to 
5, as described in Table 3. 

TABLE 3 





Quality ratiniz on a 1 to 5 scale. 




Rating 


Impairment 


Quality 


5 


Imperceptible 


Excellent 


4 


Perceptible, not annoying 


Good 


3 


Slightly annoying 


Fair 


2 


Annoying 


Poor 


1 


Very Annoying 


Bad 



The quality rating on this scale is obtained using the 
normalized conversion described in Eq. (2): 



Q = 



(2) 



l + NE 



where Q is the quality rating and E is the measured distor- 
tion. N is a normalization constant. This constant is usually 
chosen so that a known reference distortion maps to a known 
quality rating. In the case of a perceptual metric as MPQM, 
the choice of N has to be made on the basis of the vision 
model. For one embodiment, N has been estimated in the 
following way. Assume a sequence that only has an error of 
one jnd in only one channel, at a single pixel of a single 
block. This is the smallest error that could theoretically be 
perceived. Hence the quality rating of that sequence should 
be extremely close to the highest quality level. We consid- 
ered that such an error would yield a quality rating of 4.99 
and solved for N in Eq. (2). The value of N turns out to be 
0.623. 

Thus, an automatic testing method and device has been 
described that can test a video sequence coder/decoder 
system and either assess the quality of decoded sequences or 
rate the fidelity of the coding chain. 

The device produces synthetic test patterns that induce the 
appearance of known artifacts so as to be able to track and 
evaluate them. Based on this evaluation, it can rate the 
performance of the system with a good correlation with 
human judgment. 

The many features and advantages of the invention are 
apparent from the written description and thus it is intended 
by the appended claims to cover all such features and 
advantages of the invention. Further, because numerous 
modifications and changes will readily occur to those skilled 
in the art, it is not desired to limit the invention to the exact 
construction and operation as illustrated and described. 
Hence, all suitable modifications and equivalents may be 
resorted to as falling within the scope of the invention. 

What is claimed is: 
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1. A method for testing at least a portion of a coding chain, nizer to receive synchronization information regarding a first 
the coding chain having an encoder, a transmission channel video sequence of synthetic patterns to be transmitted via the 
and a decoder, the method comprising the steps of: coding chain; and 

receiving synchronization information regarding a first a first test pattern generator to generate a second video 

video sequence of synthetic patterns to be transmitted 5 sequence of synthetic patterns synchronously with the 

via the coding chain; and first sequence using the synchronization information, 

generating a second video sequence of synthetic patterns the second sequence for comparison to the first 

synchronously with the first sequence using the syn- sequence to thereby test the portion of the coding chain, 

chronization information, the second sequence for 11. An apparatus as set forth in claim 10, wherein: 

comparison to the first sequence to thereby test the 10 t he synchronizer further receives pattern sequence infor- 

portion of the coding chain. mation regarding the first video sequence of synthetic 

2. A method as set forth in claim 1, the method further patterns to be transmitted via the coding chain; end 
comprising the steps of: the fifSt fcst paUern generator generates the second video 

receiving pattern sequence information regarding the first JS sequence of synthetic patterns using the pattern 

video sequence of synthetic patterns to be transmitted sequence information. 

via the coding chain; and 12. An apparatus as set forth in claim 10, the apparatus 

generating the second video sequence of synthetic pat- further comprising an evaluation module to compare the 

terns using the pattern sequence information. second sequence to the first sequence. 

3. A method as set forth in claim 1, the method further 2 o 13. An apparatus as set forth in claim 10, the apparatus 
comprising the step of comparing the second sequence to the further comprising the decoder to decode the first video 
first sequence. sequence of synthetic patterns transmitted via the coding 

4. A method as set forth in claim 1, the method further chain. 

comprising the step of decoding the first video sequence of 14. An apparatus as set forth in claim 10, the apparatus 

synthetic patterns transmitted via the coding chain. 2 5 farther comprising: 

5. A method as set forth in claim 1, the method further the transmission channel to transmit the first video 
comprising the steps of: sequence of synthetic patterns; and 

transmitting the first video sequence of synthetic patterns t he decoder to decode the first video sequence of synthetic 

via the transmission channel; and patterns, 

decoding the first video sequence of synthetic patterns. 30 15. An apparatus as set forth in claim 10, the apparatus 

6. A method as set forth in claim 1, the method further further comprising the encoder to encode the first video 
comprising the step of encoding the first video sequence of sequence of synthetic patterns to be transmitted via the 
synthetic patterns to be transmitted via the coding chain. coding chain. 

7. A method as set forth in claim 1, the method further 16. An apparatus as set forth in claim 10, the apparatus 
comprising the step of generating pattern sequence infor- 35 further comprising a second test pattern generator to gener- 
mation regarding the first video sequence of synthetic pat- ate pattern sequence information regarding the first video 
terns to be transmitted via the coding chain. sequence of synthetic patterns to be transmitted via the 

8. A method as set forth in claim 1, the method further coding chain. 

comprising the step of generating the first video sequence of 17. An apparatus as set forth in claim 10, the apparatus 

synthetic patterns to be transmitted via the coding chain. 40 further comprising a second test pattern generator to gener- 

9. A method as set forth in claim 1, the method further ate the first video sequence of synthetic patterns to be 
comprising the step of generating complex texture images as transmitted via the coding chain. 

part of the first video sequence of synthetic patterns to be 18. An apparatus as set forth in claim 10, herein complex 

transmitted via the coding chain to test buffer occupancy texture images are generated as part of the first video 

level induced artifacts. 45 sequence of synthetic patterns to be transmitted via the 

10. An apparatus for testing at least a portion of a coding coding chain to test buffer occupancy level induced artifacts, 
chain, the coding chain having an encoder, a transmission 

channel and a decoder, the apparatus comprising: a synchro- ***** 
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